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This intervention report presents findings from a systematic review of ACT Aspire™ conducted using 
the WWC Procedures and Standards Handbook (version 3.0) and the Transition to College review 
protocol (version 3.2). No studies of ACT Aspire ™ that fall within the scope of the Transition to College 
review protocol meet What Works Clearinghouse (WWC) group design standards. Because no studies 
meet WWC group design standards at this time, the WWC is unable to draw any conclusions based 
on research about the effectiveness or ineffectiveness of ACT Aspire ™ on high school and college 
students. Research that meets WWC design standards is needed to determine the effectiveness or 
ineffectiveness of this intervention. 

Intervention Description 1 

The ACT Aspire™ system provides a longitudinal, systematic approach for assessing and monitoring students’ 
preparation for high school studies and readiness for college and career. ACT Aspire ™ includes assessments for 
students from grade 3 through early high school in five subject areas: English, mathematics, reading, science, 
and writing. The system uses a standard scoring system that measures progress through each grade level and 
culminates with the ACT® college admissions test. The ACT Aspire™ system includes a variety of reporting features 
that permit schools to track individual student progress and examine trends for groups of learners. 

An earlier version of the program, the Educational Planning and Assessment System (EPAS®), included assessments 
for students in grades 8 and 9 to measure preparation for high school studies (EXPLORE®), grade 1 0 to measure 
preparation for college and the workplace (PLAN®), and grades 1 1 and 12 to measure readiness for life after high school 
(the ACT®). ACT began phasing out the use of EPAS® in 201 4 and replaced it with the new ACT Aspire™ system. 

Research 2 

The WWC identified 18 studies of ACT Aspire™ for high school and college students that were published or released 
between 1994 and 2016. 

• Two studies are within the scope of the Transition to College review protocol but do not meet WWC group 
design standards because baseline equivalence of the analytic intervention and comparison groups is necessary 
and not demonstrated. One study (ACT, 2013) is a quasi-experimental design that compared postsecondary 
enrollment, performance, and completion rates of students who participated in ACT’s early monitoring activities 
(i.e., taking the EXPLORE® and/or the PLAN®) and students who did not. The primary sample included 33,51 0 
ACT-tested high school graduates in Oklahoma who immediately enrolled in an in-state college in the fall after 
high school graduation in 2001 or 2002. Because eligible outcomes did not have a natural pretest (e.g., college 
completion), the study authors needed to demonstrate baseline equivalence on academic achievement and 
student socioeconomic status, as required by the Transition to College protocol (version 3.2), but did not. The 
second study (Williams & Noble, 2005) is a quasi-experimental design that compared ACT® performance in 
high schools that implemented the PLAN® between 1994 and 2000 (and consistently tested all sophomores 
once implementation began) and schools that never used the PLAN® between 1994 and 2003. Outcomes were 
measured using 3-year school-level average ACT® scores from 2001, 2002, and 2003. Baseline equivalence data 
were presented from prior cohorts of tenth graders who took the ACT®. WWC standards require that baseline 
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equivalence for this cluster quasi-experimental design demonstrate baseline equivalence on an earlier, adjacent 
cohort used in the impact analysis. The study provided baseline data for intervention schools that included the 
adjacent cohort and several other baseline cohorts. The study provided baseline data for comparison schools 
that did not include an adjacent cohort; therefore, the study failed to establish baseline equivalence. 

• Eleven studies are out of the scope of the Transition to College review protocol because they have an ineligible 
study design. 

• Five studies are out of the scope of the Transition to College review protocol for reasons other than study design. 
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Endnotes 

1 The descriptive information for this intervention comes from a publicly-available source: the developer’s website (http://www.act. 
org, downloaded January 2017). The What Works Clearinghouse (WWC) requests that developers review the intervention description 
sections for accuracy from their perspective. The WWC provided the developer with the intervention description in February 2017; 
however the WWC did not receive a response. Further verification of the accuracy of the descriptive information for this intervention is 
beyond the scope of this review. 

2 The literature search reflects documents publicly available by March 2016. Reviews of studies in this report used the standards from 
the WWC Procedures and Standards Handbook (version 3.0) and the Transition to College review protocol (version 3.2). The evidence 
presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2017, May). Transition 
to College intervention report: ACT Aspire™. Retrieved from https://whatworks.ed.gov 
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Glossary of Terms 


Attrition Attrition occurs when an outcome variable is not available for all subjects initially assigned to 
the intervention and comparison groups. If a randomized controlled trial (RCT) or regression 
discontinuity design (RDD) study has high levels of attrition, the validity of the study results 
can be called into question. An RCT with high attrition cannot receive the highest rating of 
Meets WWC Group Design Standards without Reservations, but can receive a rating of Meets 
WWC Group Design Standards with Reservations if it establishes baseline equivalence of the 
analytic sample. Similarly, the highest rating an RDD with high attrition can receive is Meets 
WWC RDD Standards with Reservations. 

For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and individuals 
leave the group. If a single-case design does not meet minimum requirements for phases and 
data points within phases, the study cannot receive the highest rating of Meets WWC Pilot 
Single-Case Design Standards without Reservations. 

Baseline a point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at baseline. 
In a single-case design experiment, the baseline condition is a period during which participants 
are not receiving the intervention. 

Clustering adjustment An adjustment to the statistical significance of a finding when the units of assignment 

and analysis differ. When random assignment is carried out at the cluster level, outcomes 
for individual units within the same clusters may be correlated. When the analysis is 
conducted at the individual level rather than the cluster level, there is a mismatch between 
the unit of assignment and the unit of analysis, and this correlation must be accounted for 
when assessing the statistical significance of an impact estimate. If the correlation is not 
accounted for in a mismatched analysis, the study may be too likely to report statistically 
significant findings. To fairly assess an intervention’s effects, in cases where study authors 
have not corrected for the clustering, the WWC applies an adjustment for clustering when 
reporting statistical significance. 


Confounding factor a confounding factor is a component of a study that is completely aligned with one of the study 
conditions, making it impossible to separate how much of the observed effect was due to the 
intervention and how much was due to the factor. 


Design The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence 
of an intervention (single-case design). Designs eligible for WWC review are randomized 
controlled trials, quasi-experimental designs, regression discontinuity designs, and single- 
case designs. 

Effect Size The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 


Eligibility a study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

Equivalence a demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Glossary of Terms (continued) 


Extent of evidence An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence 
categories: small and medium to large. 

• small: includes only one study, or one school, or findings based on a total sample 
size of less than 350 students and 14 classrooms (assuming 25 students in a class) 

• medium to large: includes more than one study, more than one school, and 
findings based on a total sample of at least 350 students or 14 classrooms 

Gain scores The result of subtracting the pretest from the posttest for each individual in the sample. 

Some studies analyze gain scores instead of the unadjusted outcome measure as a method 
of accounting for the baseline measure when estimating the effect of an intervention. The 
WWC reviews and reports findings from analyses of gain scores, but gain scores do not 
satisfy the WWC’s requirement for a statistical adjustment under the baseline equivalence 
requirement. This means that a study that must satisfy the baseline equivalence requirement 
and has baseline differences between 0.05 and 0.25 standard deviations Does Not Meet 
WWC Group Design Standards if the study’s only adjustment for the baseline measure was 
in the construction of the gain score. 


Group design 

Improvement index 


A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 

Along a percentile distribution of individuals, the improvement index represents the gain or 
loss of the average individual due to the intervention. As the average individual starts at the 
50th percentile, the measure ranges from -50 to +50. 


Intervention An educational program, product, practice, or policy aimed at improving student outcomes. 


Intervention report a summary of the findings of the highest-quality research on a given program, product, 

practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 


Multiple comparison An adjustment to the statistical significance of results to account for multiple comparisons 
adjustment in a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 
the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 


Outcome domain a group of closely-related outcomes. A domain is the organizing construct for a set of related 
outcomes through which studies claim effectiveness. 


Quasi-experimental 
design (QED) 


A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 
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Glossary of Terms (continued) 


Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Regression 
discontinuity design 
(RDD) 

Single-case design 
Standard deviation 

Statistical significance 

Study rating 


Substantively important 
Systematic review 


A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

For group design research, the WWC rates the effectiveness of an intervention in each domain 
based on the quality of the research design and the magnitude, statistical significance, and 
consistency in findings. For single-case design research, the WWC rates the effectiveness of an 
intervention in each domain based on the quality of the research design and the consistency of 
demonstrated effects. 

A design in which groups are created using a continuous scoring rule. For example, 
students may be assigned to a summer school program if they score below a preset 
point on a standardized test, or schools may be awarded a grant based on their score 
on an application. A regression line or curve is estimated for the intervention group and 
similarly for the comparison group, and an effect occurs if there is a discontinuity in the two 
regression lines at the cutoff. 

A research approach in which an outcome variable is measured repeatedly within and across 
different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result 
of chance rather than a real difference between the groups. The WWC labels a finding 
statistically significant if the likelihood that the difference is due to chance is less than 
5% (p < .05). 

The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets l/l /WC Design Standards without Reservations, Meets l/l/l/l/C Design Standards with 
Reservations, or Does Not Meet l/l/l/l/C Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 

A review of existing literature on a topic that is identified and reviewed using explicit methods. 
A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their findings; 
4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 


ACT Aspire™ May 2017 


Page 8 


WWC Intervention Report 



Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Abt Associates under contract ED-IES-16-C-0024. 
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